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This is the fourth of a series of general, special, and varied reports 
of the Project: Conditions, Requirements, and Variables Affecting the 

Quality of Complex Learning Mediated by Instructional Materials , This spe- 
cial report is, as its title indicates, the "Description of a Practical 
Procedure for Assessing Instructional Film and Television Programs." 

Ideally, the quality or effectiveness of instructional materials is 
measured in terms of the degree to which they stimulate, arouse, shape, 
and investigate learned changes in human behavior. It has been argued in 
General Report Number I of this Project that a great many sets of differ- 
ent conditions affect learning and that the -“intrinsic or potential quality 
of the immediate stimulus materials, such as recordings on films or video- 
tapes, represent important but narrow-band influences in learned behavioral 
changes. This line of argument holds, furthermore, that the general qual- 
ity of such materials, even though accurately measured, cannot be expected 
to have very high correlations or close relations with affected learned 
behavior. 

It is important, nevertheless, within the area of instructional mate- 
rials production, selection, procurement, and use to have good judgmental 
evaluations of the stimulus materials, factors, and components which affect 
learning behavior. Furthermore, the practical procedures for assessing 
the factors of quality of instructional materials must be reasonably eco- 
nomical. Accordingly, however desirable it is to determine their effec- 
tiveness by actual tests of the extent or decree to which they, and they 
alone, influence learned behavior in the direction of instructional objec- 
tives, this costly and time-consuming procedure is simply not practical 
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under most present conditions of producing, selecting, and procuring film 
and television programs . Perhaps this approach will become practical when 
the nation has developed a system of applied theory of instruction and 
implemented its use with the necessary funds, trained people, and networks 
of production facilities superbly designed for the purpose of producing 
and assessing instructional materials of high quality. 

Practices and procedures currently used for evaluating instructional 
television and film programs extensively use human judgments, both of indi- 
viduals and judging panels. The judgmental schedule form is not new; it 
has been under occasional development for twenty years and, as this report 
shows, it needs further improvement. Program series for National Educa- 
tional Television and for The National Center for School and College Tele- 
vision are selected and approved on the basis of the evidence from human 
judgments. In this, the assessment and evaluation procedures correspond 
more closely to those used in the fine arts than those used in the field 
of the sciences of learning. In reality, instructional program materials 
are often selected merely from written descriptions and from limited 
sample viewing rather than by means of systematic assessment with direct 
observations of special judging panels and adequate sampling of programs. 
Thus, the practical and the useful procedures should be accepted for 
development and improvement until the ideal operational evaluations can 
be achieved. 

During the discussions at the twelve seminars conducted by this Project 
on the topic, Factors Affecting the Quality of Instructional Materials,* 
it became evident again and again that references needed to be made to 

*General Report No. 1 of this Project 
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direct observations, analyses, and assessments of instructional units and 
programs. Accordingly, the Project staff undertook to develop and test 
once again the form which is described in this paper and which is intended 
to serve as a guide for judgments about the estimated effectiveness of 
instructional materials. The hope was, also, to identify the specific 
factors that contribute to the general characteristic of quality in pro- 
grams already judged to be good or superior. 

A small panel of media specialists was selected and given the respon- 
sibility for conducting the evaluation accordingly. It is noteworthy that 
these specialists who invested the time required to test, study, and 
report on the procedure found the task rewarding and well worth the effort. 
Use of the evaluation form has training values. 

The judging of athletic performances and of exhibitions of art has 
been developed into human precision performances. Is it not possible to 
develop similar judgmental precision about the quality factors of instruc- 
tional materials? 



C. R. Carpenter 
Project Director 



University 
August 12, 



Park, Pennsylvania 

1968 
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PROCEDURE 



Since the development of high-quality educational programs is of 
great importance to this Project, the Steering Committee decided that a 
number of existing educational television and film programs considered 
to be exceptionally ■well produced should be studied in order to determine 
which elements in each contribute to the estimated high quality of the 
program. Two other tasks were undertaken: first, to test once again and 

make suggestions for the improvement of the form for evaluating film and 
television programs; second, to study the needs for developing related 
procedures for using the form in practical conditions of the selection, 
procurement, and production of instructional programs. 

Nominations of such programs were collected from participants of the 
Project Harvest Seminars as well as from members of the Steering Committee. 

Nominations included the following television programs: 

CODE FOR CLASSIFICATION OF INSTRUCTIONAL TELEVISION PROGRAMS AND FILMS 



Age Level 

1 Informal Education, Children 

2 Pre-school 

3 Primary (1-3) 

4 Intermediate (4-6) 

5 Junior and senior high (7-12) 

6 College and university 

7 Adult - Instructional and Training 

8 Adult - General 



Subject Area 

A Literature J Current Events 

B Science K Health 

C Art L Physical Education 

D Music M General 

E Language N Theatre Arts 

F Math 

G History 

H Geography 

I Driver Education 
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INSTRUCTIONAL TELEVISION PROGRAM NOMINATIONS 



Name of Program 


Produced by 


Age 

Level 


Subject 

Area 


Mister Roger's Neighborhood 


WQED, Pittsburgh 


I 


M 


Cover to Cover 


WETA, Washington 
Jay Robbins 


4 


A 


Pocketful of Fun 


21" Classroom 
Boston 


2 


CD 


Let's Investigate 


ft ft 


4 


B 


Children's Literature 


University of 
Nebraska 

Dolores & Bob Dudley 


3 


A 


Biology series 


Dade County School 
System, Miami, Fla. 


5 


B 


Meaning in Art 


WPSX, Penn State 


3 


C 


Roundabout 


WETA, Washington 


2 


M 


Stepping into Rhythm 


WVIZ, Cleveland 


3 


D 


AIL About You 


21" Classroom 


3 


K 


Tell Me a Story 


WQJD, Pittsburgh 


3 


A 


You and Eye 


KQJED, San Francisco 


4 


C 


Wordsmith 


KQED, San Francisco 


3 


E 


Meet the Arts 


21" Classroom 


3/4 


c/d 


Patterns in Arithmetic 


University of Wisconsin 


4 


F 


World of Change 


21" Classroom 


4 


B 


Americans AIL 


KRMA, Denver 


4 


G 


Geography for Grade 4 


KRMA, Denver 


• 4 


G 


Sportsmanlike Driving 


South Carolina ETV 


5 


I 


Franklin to Frost 


University of Michigan 


5 


A 


Par Ions Francais 


Heath de Rochemont 


4 


E 



3 



Name of Program 


Produced by 


Age 

Level 


Subject 

Area 


Meteorology 300 


WPSX, Penn State 


6 


B 


Sons and Daughters 


WITF, Hers hey 


5 


K 


Places in the News 


WNYE, New York 


k 


J 


Secondary Developmental 
Reading 


WQJSD, Pittsburgh 


5 


E 


Pennsylvania History and 
Government 


WITF, Hers hey 


5 


G 


Alive and About 


WEDH, Hartford 


k 


B 


American Historic Shrines 


TOT, New York 


k 


G 


Health — Your Decision 


WETA, Washington 


5 


K 


Come Read to Me a Poem 


WNYE, New York 


k 


A 


Many Sounds of Music 


WPSX, Penn State 


5 


D 


Adapted Physical Education 


WPSX, Penn State 


7 


L 


American Civilization 


University of Texas 


6 


G 



by its Interpreters 



o 
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FILM NOMINATIONS 



The Educational Film Library Association provided a list including 
the "Best Ten" educational films produced since 1959* These were selected 
by the membership of the Association. The following list also includes 
additional films which received a large number of votes: 

Nominations included the following films: 



Name of Film 


Producing Company 


Distributing 

Agency 


Age 

Level 


Subject 

Area 


AN OCCURRENCE AT 
OWL CREEK BRIDGE 


University of South- 
ern California 


University of 

Southern 

California 


5,6,8 


A,M,N 


THE RED BALLOON 


Albert Lamorisse 


Brandon Films 


1,2,3, 

4,5,6, 

8 


N,C 


SEARCH FOR 
ULYSSES 


Carousel 


Carousel 
Films, Inc. 


1,5,6, 

8 


A,G 


UNIVERSE 


National Film Board 
of Canada 


McGraw-Hill 


1,5,6 


B,N 


THE GOLDEN FISH 


Columbia Picture 
Corporation 


Brandon Films 


1,3,4 


A,C 


DISCOVERING 

COLOR 


Film Associates of 
California 


Film Associates 
of California 


3,4 


C 


RAINSHOWER 


Dimension Films 


Churchill Films 


1,3,4, 

5 


A,C 


LOUVRE 


National Broadcast- 
ing Company, TV 


Encyclopedia 
Britannica Films 


5,6,8 


C,G 


NIGHT AND FOG 


Argus, Inc. 


McGraw-Hill 


6,8 


N,G 


PADDLE TO THE. 
SEA 


National Film Board 
of Canada 


McGraw-Hill 


3A 


A,H,M 



Name of Film 


Producing Company 


Distributing 

Agency 


Age Subject 

Level Area 


NEW YORK, 
NEW YORK 


Francis Thompson 


Campus only 


4,5,6, A,C,N 
8 


CITY OF GOLD 


National Film Board 
of Canada 


McGraw-Hill 


5,6,8 A,G,N 


PRINTS 


AC I Productions 


AC I Productions 


4,5,6, C 
8 


FISHING ON THE 
COAST OF JAPAN 


International Film 
Foundation 


International 
Film Foundation 


4,5,6, G,C,D, 
8 


SYMMETRY 


Contemporary Films 


McGraw-Hill 


5,6,7, B,C,F,N 
8 


PIGS 


Churchill Films 


Churchill Films 


1 , 2 , 3 , C,A 
>*,5,8 


HAILSTONES & 
HALIBUT BONES 


Fine Arts Produc- 
tions 


Sterling Educa- 
tional Films 


1 , 2 , 3 , C,A 
4,5,8 


CHILDREN OF THE 
WAGON TRAIN 


Young America Films 


McGraw-Hill 


4,5 G 


SETTLING THE 
GREAT PLAINS 


McGraw-Hill 


McGraw-Hill 


4,5 G 


HUMAN 

REPRODUCTION 


McGraw-Hill 


McGraw-Hill 


4,5 K,L 


TRUE STORY OF 
THE CIVIL WAR 


Camera Eye Pictures 


McGraw-Hill 


5,6,8 G,M 


LEARNING 
DISCRIMINATIONS 
AND SKILLS 


McGraw-Hill 


McGraw-Hill 


6,7 B 


PHOEBE: STORY 

OF PREMARITAL 
PREGNANCY 


National Film Board 
of Canada 


McGraw-Hill 


5,6,7 K,L 
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FINAL SELECTIONS OF TELEVISION AND FILM PRODUCTIONS 



From the nominated lists of titles of the instructional television 
and film programs, the following subjects were selected for study and 
evaluation: 



Series 

All About You 
World of Change 
Meaning in Art 
Cover to Cover 
Patterns in Arithmetic 
Children’s Literature 



TELEVISION 



FILM 



Program 

Getting the Message 
Motion 

Do You Recognize? 
Wind in the Willows 
Arithmetic I 
Tobias 



Pigs 

Symmetry 

True Story of the Civil War 
Universe 

Hailstones and Halibut Bones 



The evaluation form or judgmental schedule that was used was first 
developed by the Penn State Instructional Film Research Program during the 
1950’s and has been modified many times in many ways since it was first 
developed. The latest form was being used by The National Center for School 
and College Television. This form was modified by the Project’s Steering 
Committee and circulated to seminar participants and consultants for sug- 
gested modifications. The final form used for testing in the Project 
appears on page 30 of this Report. 

ERIC 



METHOD OF ASSESSMENT 



The Steering Committee named three faculty members of The Pennsylvania 
State University as a subcommittee for conducting the test evaluations * 

They were Marlowe Froke, Chairman, Director, Division of Broadcasting;* 

E. Arthur Hungerford, Associate Professor, Speech;* Dennis Sherk, In-School 
Service Coordinator.* 

The following instructions were given to the subcommittee: 

1. At each evaluation session, one or more members of the subcommittee 
should be present. 

2. All three members of the viewing panel should use the same form 
and agree on each item. For each program viewed, there should be 
only one completed form. 

The Steering Committee agreed to add Donald Johnson, Associate Direc- 
tor, Division of Instructional Services,* to the subcommittee and several 
other interested persons were invited to attend each of the evaluating ses- 
sions. They were Thurston Reeves, Director, Audio-Visual Services;* 

Daniel Smith, Executive Director, Allegheny Educational Broadcast Council; 
and Robert Dudley, Assistant Director, Division of Broadcasting.* 

Evaluation sessions were held on May 27, 29, and 31 , 1968 . All sub- 

* 

committee members and evaluators attended the first session only. Mr. Reeves 
was absent from the remaining two sessions. 

At the first subcommittee meeting, it was decided that this would be 
the procedure for evaluation: 

*The persons are members of The Pennsylvania State University faculty. 
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1. Identification of program. 

2. Viewing of program. 

3. Completion of the evaluation form by each evaluator. 

4. Preparation of the composite evaluation form by evaluators 
based on their individual evaluations. 

For purposes of interpretation, the results of the judgments were 
tabulated. Point values ranging from 4 to 0 were assigned to the five- 
point scale in questions 1 through 13, and 15 through 19. Question ik 
was a two-point scale with values of 4 and 0 assigned. DMA— Does Not 
Apply — was not included in the scale; a separate tabulation was made of 
these responses. 

Because of the broad range of applicability of the evaluation instru- 
ment to the specific television and film programs, no statistical signifi- 
cance ether than as a means for general observations could be given to the 



tabulation. 



TABULATION OF RESULTS OF EVALUATION 
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EXPLANATION OF TABULATION 



A summary of the tabulation is as follows: The left-hand listing 

identifies the title of the television or film program, whether it was a 
television program (T) or a film program (F), and the number of evaluators. 
Columns numbered 1 through 19 refer to the questions in the evaluation 
instrument. For all programs the top number in the ^ listing is the number 
of points scored by each program on the question with the point value on 
the scale multiplied by the number of evaluators ranking the program. The 
bottom number is the number of DBA — Does Not Apply — responses. 

In column 19 an effort was made to show an average score for each pro- 
gram on question 19 which was, "What is your overall evaluation of the 
lesson?" The average was determined by taking the total points scored by 
the program on question 19 and dividing the number of evaluators. 

The next to the last column reports the total points scored above the 
line, and the total number of DNA responses below the line. The last col- 
umn is the average number of points for each program. It was arrived at 
by dividing the total number of points scored on the questions by the number 
of evaluators. 

The asterisks following some numbers indicate a reservation about the 
applicability of this item of the form for the specific program being evalu- 
ated. For example, the program, "Symmetry," scored 0 points on questions 
6, 13j and l4. Question 6 concerned vocabulary level, question 13 concerned 
personality of the teacher, and question l4 concerned the length of time 
of the teacher on the screen. In the program, "Symmetry," there was no 
spoken voice, and a teacher never appeared on camera. It is probably 
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impossible to develop a general form for structuring judgments -which is 
equally applicable to all kinds of specific programs. 
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SUMMARY OF PANEL DISCUSSION 

Generally, the programs assessed which received the highest scores 
were highly visual and of good production quality. It was discussed at 
length, however, whether such qualities necessarily meant that the pro- 
grams were of high quality as means for learning . For instance, Arith- 
metic I is more effective as a learning device than some of the other 
television programs which were evaluated, even though its production qual- 
ity was below average . The instructor in Arithmetic I appeared before the 
camera with conventional classroom aids such as the blackboard for the 
entire length of the program. The simplicity of the presentation, however, 
did not detract from the rich flow of substantive concepts that were pre- 
sented. 

There was general agreement that the overall technical and produc- 
tion quality of instructional television and film has increased signifi- 
cantly in the last four or five years. Part of the explanation for this 
improvement can be attributed to the development of organizations for iden- 
tifying and distributing good instructional television productions. Some 
of these are the Great Plains Instructional Television Library, the Eastern 
Educational Network Instructional Service, and the National Center for 
School and College Television. 

A lack of understanding of the uses of instructional television and 
film sometimes leads to a misunderstanding of quality production. A pro- 
• gram or series should include in its statement of objectives information 
about the educational strategies of its use. For instance, instructional 
/ \ television can be used effectively as a distribution mechanism for instruc- 

tion presented through another medium, i.e., the conventional classroom. 

ERIC 
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If it is used specifically in this manner, the output should not be evalu- 
ated against other different uses of instructional television. Used as 
means of distribution , instructional television could be rated high in 
quality. The potential uses of instructional television and the varied 
production elements which it includes make it impossible to evaluate tele- 
vision as a narrow-band medium. 

Generally, the production values of the film units were judged to be 
superior to those of the instructional television units. An effort should 
be made to identify more clearly the ’’art” of instructional television as 
distinct from the "art" of film. Once the unique characteristics of 
instructional television are distinguised from those of film, there might 
be laid a foundation upon which to build instructional television programs 
which are uniquely appropriate in education. Two significant characteris- 
tics of instructional television can be identified: (l) its use as a dis- 

tribution device for large bodies of material, e.g., the hardware for dis- 
tributing materials from one point to another; and (2) its use as a pro - 
gram production device for instructional materials that can be distributed 
both by television and by other means. 

Although instructional television production personnel can learn a 
great deal from their counterparts in film production, instructional tele- 
vision has the potential for becoming firmly established in the educational 
system due to the added features of supplementary instructional program 
materials and in-service teacher training. On the elementary level, a 
major system for the use of instructional televised units has been developed. 
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RECOMMENDATIONS 

The panel made the following recommendations: A system of evaluation 
for an instructional television and film program should include separate 
assessments by (l) panels of audience for which the program is intended, 
(2) panels of educational specialists, (3) panels of content specialists, 
(4) panels of production specialists, and (5) panels of media specialists. 
The judgmental form used should be adapted and differentiated for these 
and other specific groups of evaluators. 
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IQ- TV FORM FOR EVALUATING THE INSTRUCTIONAL EFFECTIVENESS 
OF FILMS OR TELEVISION PROGRAMS 

NOTE : THIS IS THE FORM THAT WAS USED BY THE PANEL. IN THE BLOCK AFTER 
EACH QUESTION, THE COMMENTS BY THE PANEL ARE NOTED. WHERE THE PANEL RECOM- 
MENDED CHANGES IN THE QUESTION, THE REWORDED QUESTION FOLLOWS. \ 

Name of film or 

Name/number of program 

Title of Series ________________________ 

Produced by 

Distributed by 
Date of viewing 



This form has been designed to study the factors and elements in a unit 
of instructional material which contribute most significantly to its 
achievement of excellent quality. For the purposes of this evaluation, 
quality is defined as those factors which produce the desired behavioral- 
changes in the target population. 

Please circle the term which represents your best judgment of the 
degree to which the program satisfies each criterion. Feel free to add 
any comments which will help to describe the reasons for your evaluation. 
If you believe the criterion does not apply, please encircle DNA. 



I. OBJECTIVES 

1. Are the instructional objectives as stated or implied in the 
lesson clear to the viewer? 

Very clear Clear Adequate Unclear Very unclear DNA 

What are the objectives? . 



How are they stated? By whom? 




COMMENTS: 



PANEL COMMENTS: 



The instructional objectives were not clearly stated 
in most of the programs reviewed. This led to difficulty 
in answering all parts of this question. 

It was suggested that program objectives would be more 
explicit if titles of programs clearly reflected the princi- 
pal objectives of the program or program unit. Statements 
of objectives could also be included in the beginning of 
units. Although this procedure may not be artistically 
appropriate, it would have educational merit. 

— t 

2. Does the content of the program relate closely to the main objec- 
tives, or are there many irrelevancies? 

Very Some Many 

closely Closely Adequately irrelevancies irrelevancies DNA 

COMMENTS : 



PANEL COMMENTS: 

Unless program objectives are clearly stated, it is difficult 
to judge the relation of the content to the objectives. 



CONTENT 

3. Does the amount of time taken to develop each concept, procedure, 
or example seem appropriate or inappropriate for the intended 
audience? 

Highly ^ Somewhat Highly 

appropriate Appropriate Acceptable inappropriate inappropriate DNA 



COMMENTS: 
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PANEL COMMENTS: 

The answer to this question could come only from the 

i 

j audience for which the program was intended. This question 

j 

I gives added emphasis to the necessity for the pretesting of 

j 

. instructional materials and retesting during production. 
Information on effects on learning should be available to 
panels of judges. 



4. Is the content organized and so structured as to facilitate 
learning? 

Very well Well Adequately Poor Very poorly DNA 

COMMENTS : 



PANEL COMMENTS: 

This is a good question for learning specialists and 
producers . 



5. Is the material based on expert up-to-date professional informa- 
tion? 

Contains Contains 

latest Very Adequately obsolete Very 

knowledge up-to-date up-to-date information obsolete DNA 

COMMENTS: 



PANEL COMMENTS : 

Some of the programs had purely artistic value, e.g., 

Tobias and Wind in the Willows . In these television pro- 
grams, professional information is not a factor to be considered. 
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6. Is the vocabulary level appropriate for the intended audience? 

Highly Very Appropriate In- Very in- 
appropriate appropriate level appropriate appropriate DNA 

COMMENTS: 



PANEL COMMENTS: 

This is a good question for learning specialists in 
particular and needs to be considered in all productions. 



PRESENTATION OF MATERIAL 

7. Does the presentation provide for optimum repetition of the main 
ideas? (e.g., Summaries of main points from time to time and at 
end; repetition with variation. ) 

Far too 

Optimum Adequate Some Too little little or . 

repetition repetition repetition or too much far too much DNA 

COMMENTS: 



PANEL COMMENTS: 

This question could best by answered by members of the 
target audience in a pretest or production test situation. 



8. Does the presentation effectively utilize the visual channel of 
communication? (e.g., Uses appropriate pictures, film clips, 
demonstrations, diagrams, etc.) 

Highly Above Moderately Below 

effective average effective average Ineffective DNA 



COMMENTS: 



PANEL COMMENTS: 



It is difficult to rank productions that use few 



visuals but use them effectively. 



PANEL RECOMMENDED CHANGE: | 

i 

i 

8. Does the program effectively use appropriate pictures, film clips, j 
demonstrations, diagrams, and other graphics? (Numbers and kinds of j 
visuals are not as important as the way in which they are used to sup- j 
port the instruction.) j 

Highly Above Moderately Below j 

effective average effective average Ineffective DNA 



COMMENTS: 



9. Is the visual presentation clearly perceivable by use of good 
lighting, appropriate camera shots, sharpness of details, 
pointers, suitable backgrounds, etc? 



Highly Clearly 

perceivable perceivable Acceptable 



Barely Unperceiv- 

perceivable able DNA 



COMMENTS: 



PANEL COMMENTS: 



The question is not clear as to whether emphasis should 



be placed on a technical and engineering evaluation or on a 



production and effects evaluation. 
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PANEL RECOMMENDED CHANGE: 

9. Is the video -photographic presentation clearly perceivable by use 
of good lighting, appropriate camera shots, sharpness of details, 
pointers, suitable backgrounds, etc? (This does not require a 
highly technical or engineering evaluation but rather a judgment 
as to whether or not the program or film is perceptually clear.) 

Highly Clearly Barely Unperceiv- 

perceivable perceivable Acceptable perceivable able DNA 

. COMMENTS: 



! 



10. Is the audio intelligible? 

High Low 

intelligi- Above Below intelligi- 
bility average Intelligible average bility DNA 

COMMENTS: 



PANEL COMMENTS: 

Unless a technical evaluation by audio specialists is 
desired, a five-point scale is unnecessary for this ques- 
tion. For many items of technical quality, the scale should 
be "go-no go" relative to established technical standards. 

PANEL RECOMMENDED. CHANGE: 

> 

10. Is the audio intelligible? 

Satisfactory Unsatisfactory DNA 



COMMENTS : 
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11. Is there an appropriate integration of visual and audio? 



Excellent Good 

integration integration Adequate 



Poor Very poor 

integration integration DNA 



COMMENTS: 



PANEL COMMENTS: 

This is a good question for production personnel and 
learning specialists. 



12. Does the presentation give the impression of enthusiasm, sin 
cerity and authenticity? 



Very 

sincere Sincere 



Very 

Satisfactory Insincere insincere 



COMMENTS: 



DNA 



PANEL COMMENTS : 

This asks for three different judgments. 



PANEL RECOMMENDED CHANGE: 



12. Does the presentation give the impression of authenticity? 



Authentic 



Lacks authenticity 



DNA 



COMMENTS: 
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13. Do the personality, voice, and appearance of the teacher or 
teachers add to or detract from the effectiveness of the pre- 
sentation? 

Adds Neutral in Detracts Detracts 

greatly Adds somewhat effects somewhat greatly DNA 

COMMENTS: 



PANEL COMMENTS: 

Many instructional television and film programs are 
and should be produced without the appearance of a teacher. 
Also, most instructional films use commentators. 

PANEL RECOMMENDED CHANGE: 

13. Do the personality and appearance of the teacher or teachers add to 
or detract from the effectiveness of the presentation? 

Adds Neutral in Detracts Detracts 

greatly Adds somewhat effects somewhat greatly DNA 

COMMENTS: 




QUESTION ADDED BY PANEL: 

13a. Do the characteristics and quality of the instructor’s or commen- 
tator’s voice add to or detract from the effectiveness of the pre- 
sentation? 

r 

Adds Neutral in Detracts Detracts 

greatly Adds somewhat effects somewhat greatly DNA 

COMMENTS: 



14. Does the teacher appear on camera for an appropriate amount of 
time? 

Optimum amount of time Too much or too little DNA 

COMMENTS: 
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PANEL COMMENTS: 

If a teacher is a good television teacher, an appro- 
priate amount of time for him to appear on camera might 
be the total time of the program. The question should 
include an estimate of the amount of time the instructor 
appears on the screen. 

j_ __ __J 

PANEL RECOMMENDED CHANGE: 

l4. Does the teacher appear on camera for an appropriate amount of 
time? 

Optimum amount Approximate per- 

of time Too much Too little cent age of time DNA 

COMMENTS : 



IV. LEARNER STIMULATION 

15. Are the techniques designed to provide viewer participation 
successful or unsuccessful? 

Highly Moderately Barely Partially Totally 

successful successful successful unsuccessful unsuccessful DNA 

COMMENTS : 



( ) 



PANEL COMMENTS: 

Some questions were raised as to what was meant by the 
term, participation. Interpretation ranged from the program- 
ming of responses to the factor of potential involvement. 




The term should be clarified. 
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PANEL RECOMMENDED CHANGE: 

IV. LEARNER STIMULATION 

15. Are the techniques designed to provide viewer participation 
successful or unsuccessful? (Participation means students 
using work sheets, devices, and other ways of involving them 
actively in the instruction.) 

Highly Moderately Barely Partially Totally 

successful successful successful unsuccessful unsuccessful DNA 

COMMENTS: 



l6. Does the presentation motivate the student to go beyond the 
actual lesson by further reading or study? 

Very Very 

high motivation High Adequate Low low motivation DNA 

COMMENTS : 



PANEL COMMENTS: 

The phrase, by further reading or study, is limiting. 
The results of the motivation should be included in the 
answer . 



PANEL RECOMMENDED CHANGE: 

16. Does the presentation motivate the student to do supplementary work 
and study on the problem? (if so, specify under COMMENTS what the 
learners might do.) 

Very Very 

high motivation High Adequate Low low motivation DNA 

COMMENTS: 
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V. LEARNER ACHIEVEMENT 

17. Is any testing incorporated into the presentation to measure the 
learners’ achievement? 

Occurs at 
appropriate 

intervals to Too much Too little No 

provide feedback testing testing testing DNA 

COMMENTS: 



PANEL COMMENT: 

In some instructional television lessons, questions are 
included in the production. In others, sets of questions are 
developed to be asked in situations of use. These questions 
are usually based on supplementary written materials that are 
made available before the telecast. 



PANEL RECOMMENDED CHANGE: 

V. LEARNER ACHIEVEMENT 

17. Is any testing incorporated into the presentation or presented 
by the classroom instructor to the students following the tele- 
cast to measure the learners’ achievement? (Note under COMMENTS 
how testing is included.) 

Appropriate 

testing Too much Too little No 

procedure testing testing testing DNA 



COMMENTS : 
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18. Is there a procedure for reporting the knowledge of test 
results? 

Extremely good 

feedback procedure. Moderately Adequate Poor No feedback DNA 
PANEL COMMENTS: 

Most of the instructional television lessons reviewed 
are used as enrichment or supplementary material by class- 
room teachers. This question only reflects the presence 
or lack of a direct reporting procedure incorporated into 
the television lesson. 



PANEL RECOMMENDED CHANGE: 

18. Is there a procedure for reporting the knowledge of test results? 
(Under COMMENTS, specify what type and to whom reported.) 

Yes No DNA 

COMMENTS : 



VI. GENERAL EVALUATION 

19. What is your overall evaluation of the lesson? 

Above Below 

Outstanding average Average average Very poor DNA 

COMMENTS : 



PANEL COMMENTS: 

With no exception, the overall evaluation of the unit 
related directly to the evaluations of the items identified 
earlier in the form. 
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PANEL RECOMMENDED CHANGE: 
VI. GENERAL EVALUATION 

19. What is your overall 

Above 

Outstanding average 
COMMENTS : 



20. What other criteria are applicable to this lesson? Use these 
criteria for further evaluation of the unit. 



evaluation of the unit? 



Below 

Average average Very poor 



DNA 



If possible, obtain facts on utilization, i.e., number of schools 
presently using the lesson or series, how often, etc. 

PANEL COMMENTS: 

Other criteria for evaluation are these: 

1. Conditions of use of the television program or film 

2. Effectiveness of film "grammar" 

3. Stimulation of active responsive learning 




PANEL RECOMMENDED CHANGE: 



20. "What other criteria are applicable to this unit? Use these criteria 
for further evaluation of the unit. If information is available, 
note here facts on utilization, i.e., number of schools presently- 
using the lesson or series, how often, etc. 



EVALUATED BY: 

NAME 

TITLE 

AFFILIATION 



MAILING ADDRESS 



PANEL COMMENTS ON FORM AS A BASIS FOR ASSESSMENT 



This judgmental form is valuable for determining general assessments 
of instructional television and film programs. However, highly specialized 
evaluation forms appropriate for statistical and objective evaluation 
should be developed. It was agreed that, although an important advance 
has been made with the development of the Project’s form, Evaluating 
Instructional Effectiveness of Films or Television Programs, much work 
remains to be done in the field of practical evaluation and dependable 
measurement in instructional media. Form reliabilities must be established 
and results of panel evaluation validated against actual learning gains. 
Furthermore, much more work needs to be done to develop and perfect the 
procedure for using the panel of judges method for training media people, 
for developing media teams, and for guiding production personnel. 

REVISION OF EVALUATION FORM 



The following is the revised form. Evaluating the Instructional Effec- 
tiveness of Films or Television Programs, which incorporates the changes 



recommended by the panel. 



Revised 8/12/68 



FORM FOR EVALUATING THE INSTRUCTIONAL EFFECTIVENESS 
OF FILMS OR TELEVISION PROGRAMS 



Name of film or 
Name/number of program 

Title of Series 

Produced by 

Distributed by 

Date of viewing 



This form has been designed to study the factors and elements in a unit 
of instructional material which contribute most significantly to its 
achievement of excellent quality. For the purposes of this evaluation, 
quality is defined as those factors which produce the desired behavioral 
changes in the target population. 

Please circle the term which represents your best judgment of the degree 
to which the program satisfies each criterion. Feel free to add any 
comments which will help to describe the reasons for your evaluation. 

If you believe the criterion does not apply, please encircle DNA. 



I. OBJECTIVES 

1. Are the instructional objectives as stated or implied in the 
lesson clear to the viewer? 

Very clear Clear Adequate Unclear Very unclear 

What are the objectives? - — 



How are they stated? By whom? 



COMMENTS : 



¥ 






t 1 



a 



( ) 
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2. Does the content of the program relate closely to the mam 
objectives, or are there many irrelevancies. 



Verv Some Many 

closely Closely Adequately irrelevancies irrelevancies DNA 



COMMENTS : 






II. CONTENT 



* Does the amount of time taken to develop each concept, procedure, 
3 ' ^example seem appropriate or inappropriate for the intended 

audience? 



m<rhiv Somewhat Highly 

appropriate Appropriate Acceptable inappropriate inappropriate DMA 



COMMENTS: 



4. Is the content organized and so structured as to facilitate 
learning? 

Very well Well Adequately Poor Very poorly DNA 



COMMENTS: 



5- 



Is the material based on expert, up-to-date professional informa- 
tion? 



Contains 

Contains r 

ifl+ps'h Very Adequately obsolete Ve y 

knowledge up-to-date up-to-date information obsolete 



DNA 



i 



COMMENTS: 
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III. 



(J 
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6 . Is the vocabulary level appropriate for the intended audience? 

Highly Very Appropriate In- Very in- 
appropriate appropriate level appropriate appropriate DNA 

COMMENTS : 



PRESENTATION OF MATERIAL 

7. Does the presentation provide for optimum repetition of the main 
ideas? (e.g., Summaries of main points from time to time and at 
end; repetition with variation.) 

Far too 

Optimum Adequate Some Too little little or 

repetition repetition repetition or too much far too much DNA 

COMMENTS: . - 



8 . Does the program effectively use appropriate pictures, film 
clips, demonstrations, diagrams, and other graphics? (Number 
and kinds of visuals are not as important as the way in which 
they are used to support the instruction.) 

Highly Above Moderately Below 

effective average effective average Inneffective D 

COMMENTS: 



9. Is the video-photographic presentation clearly perceivable by use 
of good lighting, appropriate camera shots, sharpness of details, 
pointers, suitable backgrounds, etc.? (This does not require a 
highly technical or engineering evaluation but rather a judgment 
as to whether or not the program or film is perceptually clear.) 

Highly Clearly Barely 

perceivable perceivable Acceptable perceivable 

COMMENTS: 



Unperceiv- 
able DNA 
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10. Is the audio intelligible? 



o 
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Satisfactory 



Unsatisfactory 



DNA 



COMMENTS : 



11. Is there an appropriate integration of visual and audio? 

Excellent Good Poor Very poor 

integration integration Adequate integration integration DNA 

COMMENTS : 



12. Does the presentation give the impression of authenticity? 



Authentic 



Lacks authenticity 



DNA 



COMMENTS: 



13. Do the personality and appearance of the teacher or teachers add 
to or detract from the effectiveness of the presentation? 



Adds Neutral in Detracts Detracts 

greatly Adds somewhat effects somewhat greatly 



DNA 



COMMENTS: 



14. Do the characteristics and quality of the instructor's or com- 
mentator's voice add to or detract from the effectiveness of the 
presentation? 



K ! 



Adds Neutral in Detracts Detracts 

greatly Adds somewhat effects somewhat greatly 



DNA 



COMMENTS: 



15. Does the teacher appear on camera for an appropriate amount of 
time? 

Optimum amount Approximate per- 

of time Too much Too little centage of time DMA 

COMMENTS: 



LEARNER STIMULATION 

1 6 . Are the techniques designed to provide viewer participation 
successful or unsuccessful? (Participation means students 
using work sheets, devices, and other ways of actively involv- 
ing them in the instruction.) 

Highly Moderately Barely Partially Totally 

successful successful successful unsuccessful unsuccessful DNA 

COMMENTS: 



17 « Does the presentation motivate the student to do supplementary 
work and study on the problem? (if so, specify under COMMENTS 
what the learners might do.) 

Very Very 

high motivation High Adequate Low low motivation DNA 

COMMENTS: 



18. Is any testing incorporated into the presentation or presented 
by the classroom instructor to the students following the tele- 
cast to measure the learners’ achievement? (Note under COMMENTS 
how testing is included.) 

Appropriate 

testing Too much Too little No 

procedure testing testing testing DNA 



COMMENTS: 
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' 19. Is there a procedure for reporting the knowledge of test 

L results? (Under COMMENTS, specify what type and to whom 

t reported. ) 

1 Yes No DNA 

l 

K 

i COMMENTS : 



VI. GENERAL EVALUATION 

20. What is your overall evaluation of the unit? 

Above Below 

Outstanding average Average average Very poor 
COMMENTS : 



DNA 




21. What other criteria are applicable to this unit? Use these 
criteria for further evaluation of the unit. If information 
is available, note here facts on utilization, i.e., number of 
schools presently using the lesson or series, how often, etc. 



I 

>' 

r 
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EVALUATED BY: 

NAME 

TITLE : 

AFFILIATION 



MAILING ADDRESS 



) 



r 
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